As you can see in the picture, I didn’t just terminate cables in a mechanical room and mount a few pieces of hardware directly to a wall. Instead I got a wall mount rack to organize everything in. The rack I chose had to meet a few base criteria:

• standard U sizing so mounting equipment would be easy
• significant number of Us to fit current and future equipment
• easy open access for setup and maintenance

• enough depth to accept the networking hardware I had chosen

  In the end I got a Tripp Lite rack that I could lag into some wall studs and fill with up to 150lbs of equipment. The one thing that is weak on this kind of solution is the cable management options. There are no cable raceways down the sides that you can use and third-party options are limited if they exist at all. If you look closely at the picture you can see that I ended up having to velcro strap the cable bundles to the rack posts and supports to be able to maintain some semblance of cabling hygiene.

  Getting the rack setup was pretty easy, but I did run into one issue that slowed me down. The rack has threaded holes for mounting, not holes for cage nuts. The threaded holes are ever so slightly smaller than those in cage nuts which made it impossible to use the bolts that came with most of the hardware I was mounting. If I were to do this again, I’d make the effort (and pay the necessary price) to get a wall rack that used cage nuts.

  In addition to mounting the rack on the wall I had the electricians (who pulled all my Cat6 cabling) put two plugs on the all behind the rack. The plugs are on separate circuits for future expansion and/or high draw equipment. As of now, I only need one but I did not want to have to watch someone pounding nails or running screws behind my rack in the future. I’ll cover the whole power system in another post, but safe to say it’s not just a bunch of wall-warts plugged into power bars.

The last pieces added to the rack were accessories; horizontal cable management, shelves, and blank face plates. Of those, probably the only truly necessary items were the shelves. I have one mounted at the top of the rack that holds all of my ISP’s hardware. I wanted to keep that crap isolated. The other shelf is mounted near the bottom of the rack and holds a collection of Raspberry Pi computers that I use for various things.

I spent a lot of time planning how to fill the rack. I didn’t want network wires running all over the place, and I didn’t want a mess of power cables either. In fact, I decided early on that I didn’t want the networking and power cables to intermingle at all. The best tool that I found for doing this was Google Sheets (or Excel if you must). The diagram on the left shows the plan that I came up with to meet those goals.

When I started the process of building this infrastructure, I figured getting the rack mounted was going to be one of the easier tasks. It turned out not to be hard, but to be quite time consuming as I searched for the best setup.

One of the things that I knew I wanted to do with this home network was run cables to as many places as possible. Some were non-negotiable:

• the office needed multiple drops (8 in the end)
• the TV areas needed multiple drops
• each of the wireless access points needed a drop

I really didn’t want to deal with pulling the cable into some of those areas. Hanging out in the attic wasn’t way up on my list of fun things. And, to be honest, I had no clue how I was going to get cables from the main house into the attached in-law suite. So I called the local electrician. I needed a couple of circuits to power the equipment in the server rack so it seemed like something that would make sense to bundle together.

This was the best option. The electrician and his helper pulled about 1700 feet of Cat6 cable for me. Some were my required cables and a few were extras that they pulled to the attics so that I had extra capacity when I need it. They spent a bunch of time in the heat of the attic. They also pulled cables into wall cavities so that it looks like the house has always had Cat6.

All of the cable drops were pulled to the area that I was going to put the server racks. All of this is pretty basic stuff, but I did do one thing different than I did in my last house; I didn’t cut the cables short on the rack end. Instead I terminated them into Cat6 female keystone jacks as close to the ends as possible.

This left a lot of extra cable by the rack. And that’s fine. Heck, in the end I had to re-terminate a handful of the cables because I screwed them up or did them poorly.

NOTE: It’s worth buying a proper punch down tool if you’re going to go do your own ends. Between my patch panel and the wall plates around the house I terminated about 50 keystones.

So what do you do with all that excess cable? You dress it of course. If you’ve never seen well dressed network cables, you need to spend a bit of time over at https://cableporn.reddit.com. I hate messy cables. It could be my desk, behind the TV, or my networking stuff, so the people of /r/cableporn have a dear place in my heart.

I dressed all the excess cables into what’s known as a “service loop”. The loop provides you with an excess of cable really close to the rack. If you have to fix terminations you have extra cable that you can use. This was something that I didn’t have at the old house and it almost bit me in the ass. So now I have a service loop attached right to the rack which gives me about 4-5 ft of extra cable.

I had so much excess cable that I ended up with a second service loop that has about 8ft of cable in it too…so I should never have short cable problems.

Having a service loop is great, but having a messy one is almost as bad as not having one at all. I bundled up all of the cables in the service loop using velcro straps. Not zip ties, velcro straps. You can easily undo them and move cables as needed. With zip ties I’d have to cut them and add new ones every time. It’s also possible to over tighten a zip tie and end up cutting into the cabling…not good. Velcro straps were the right choice based on the amount of times that I had to loosen them to move cables.

I probably could have bundled the cables better if I’d used a cable comb, but for the amount of cables that I had I didn’t think buying one was worth it.

The only other Cat6 cable work that I did was to make a bunch of short cables to go between the patch panel and the switch. Yah, I probably could have bought them and saved myself a bunch of time. I needed something to do in my evenings, and getting them the exact length that I wanted would make the front of the panel look a little better too.

So all that Cat6 cable management is great, but there are other cables in my rack area that needed to be managed too. The switches, routers, UPS, power distribution, etc all had power cables. Instead of running them all over the place to get them plugged in, I bundled them up too. I didn’t bundle them with the Cat6 though. Power cables can cause interference in unshielded network cabling. So with the Cat6 service loop attached to one side of the rack, I bundled the power cables and ran them along the other side of the rack.

So cabling was important, and maybe I went a bit overboard on it for a house. I should never have issues with the cables I have, and if I do I have extra cable in the service loops to fix it. I have an organized rack cabling solution that makes it really easy to trace and replace/fix cables as required. It’s also really easy to move things around, change ports being used, or completely remove hardware as needed.

Put some time into planning your cable needs, some effort into installing them, and some patience into organizing them. The end result will pay off.

Late last year we moved into a new house. Leaving the other house was tough, and not just because I had built out a fairly good networking solution for it. Moving into the new house meant a new internet provider and the standard installation of home networking gear that exists in (hundreds of) thousands of houses throughout North America. Not only did we have crap hardward, the installer obviously had taken the path of least resistence when running the wire from the street and locating it in the house. The best part of the installation was that the wire from the street entered into the mechanical room. The install or previous owners had also run a small mess of Cat5 to get from the mechanical room to a central location where the ISP’s router was expected to be located. From that location, they had run more wire to an “office” room in another part of the house.

This setup wasn’t a bad one, but it did have it’s limitations. The main router had to sit along a wall in the eating area of the kitchen. That router was also the single wireless access point in the house. ISP provided hardware being what it is (substandard at best), the wifi wasn’t stable and, in some locations of the house, there was no identifiable signal. The first 6 months living in the new house have been filled with a host of first world problems. But, being the first world, we had solutions at our disposal.

Past experience

Our old home had similar issues and I solved many of them, but I also introduced a few different problems along the way. At it’s roots that house used a Rosewill gigabit switch and a handful of DD-WRT powered routers. If nothing else, these devices were stable…after I solved the long-running issue of random router reboots that, ultimately, were being caused by a loose power plug.

Probably the biggest ongoing problem in that house was wifi handoffs. There was one DD-WRT wireless access point on each of the three floors in the house. If I moved from the basement to the main floor, my phone (or other device) would stay connected to the basement access point…even if the signal strength dropped to barely noticeable levels. The result was that you could be standing right next to an access point but you wouldn’t be connected to it and you’d likely be suffering from a poor connection to another access point. Definitely not a good thing in a house of highly connected and frequently moving people.

There were a number of things that I really liked about the physical setup in that house, and I knew that I needed to carry them forward to the new one. One of those was the use of a patch panel where all of the wired connections in the house terminated. Being able to easily interact with the different locations throughout the house was a godsend many times. I added more than just the network cables to the patch panel. Using a Keystone based panel I was able to have all of the Cat5e cables in the panel, the coaxial cables and the phone lines all in that one place. Interestingly, I benefited from the patch panel flexibility more times with the coax and phone lines than I did with the network cables.

One of the other nice thing was the use of a PDU in the rack. Instead of having multiple powered devices all searching for a wall plug and, ultimately, resorting to some kind of consumer power bar, I was able to nicely organize the power distribution.

The final “must have” that I learned in the old house was the use of a wall mount rack. Originally I had aspirations of using a 48U free standing rack for my networking and entertainment needs. Realistically I didn’t need that much space. It wasn’t just the vertical space of the free standing rack that was overkill, but also the depth. Most rack mount networking hardware doesn’t need more than 20 inches of depth. A good wall mount rack can easily provide the depth required. A rack also saves you from having a bunch of crappy shelves holding one-off hardware and a tangle of cables. For me, a wall-mount rack was a must-have.

I did a lot of good things in the old house. I did a number of bad things too. My only goal was making home-network-v2 better than its predecessor.

Needs

Top of the list, which I was constantly hearing about from the other people in the house, was strong, stable WiFi that covered all the areas we used around the house, the inlaw suite and the pool. The longer I waited to build this network, the more I was being reminded of how important this was to the other people in the house.

I also wanted to setup a good infrastructure base to build from. Strong hardware, good cable management and room for expansion. The expansion part was key. I have lots of plans.

Wants

There are a lot of things that I want to do with a home network. A big one is supporting my desire to build a full sensor network to track things like water spills/leaks, windows, doors, and many other things. One of the biggest issues with adding those features to a house is that the sensor devices require power. The easiest solution to that was a PoE capable switch and a full 48 ports.

I also really wanted to completely eliminate all of the ISP provided hardware. The WiFi on it sucks, it reboots randomly and I have little or no control over the patching of it. The biggest challenge here is that the ISP router also provides our TV service.

One of the goals with moving to this new house was for me to spend some more time in a workshop making sawdust. I’d love to be 100% disconnected while doing that, but the reality is that I do a lot of research/learning about woodworking online. Some internet connectivity in my shop, which is about 150 feet from the house, was pretty high up on my list.

Dreams

We all have them: plans for our “ideal” network. The platinum plated solution. I’m definitely not immune to this.

One of the things I hate the most about the default ISP provided solution is that all of the TV set-top boxes get their signals via WiFi. Obviously, the farther from the base unit, the more walls, bad weather, invisible temporary Faraday Cage’s, days that end in ‘Y’ and other things will cause instability in the TV signals. I really wanted to get rid of this and go to a wired solution so that I would know that the TVs would all work whenever I needed to binge watch Master Chef Junior The Expanse. This is not a normal way to configure this provider’s TV sytems.

Another dream was to setup the telephone wiring in a patch panel like I had done at the old house. This might seem like a simple thing to do, but you haven’t seen the wiring in the new house. It’s a 40+ year old place that has had low voltage wiring cobbled together through its life. Getting all of the different phone jacks routed back to a central location was going to be no small task.

Thinking and Planning and Thinking

I spent somewhere between 4 and 6 months formulating ideas, prioritizing them, re-thinking possible solutions and scrapping some of them. It was probably one of the best things that I could have done. Instead of rushing into any solution that looked and felt better than what was available when we moved in, I lived with what we had (much to the chagrin of the other WiFi users in the house). I didn’t just go out and buy the hardware I thought I needed. I thought through the problem area, researched options, considered if the problem really would exist, looked for alternatives and then re-thought everything.

I discovered a lot of problems that I would have overlooked if I’d moved faster. Things like the importance of having system management solutions when you’re using enterprise level hardware. I found a lot of hardware solutions that I didn’t know existed. One of those was front and back cable management trays. I also scrapped a few ideas, like running a buried strand of fiber from the house to the garage.

There were a few adjustments and ideas that did surface that I couldn’t ignore though.

Adjustments

Another problem that I struggled to solve was how I could monitor people who came to the door of the inlaw suite. It is situated such that I can’t see the door or the driveway from any part of the main house. I looked at some IoT-ish video doorbells, but none of them gave me the warm and fuzzies. One night I went down a rabbit hole around IP based video monitoring systems. Not only could I see who was knocking at the door, but I could also put some eyes on other parts of the yard like the detatched garage and the pool. IP based video was not just going to get added to the list or requirements, but it was going to apparear at the “want” level.

I read a lot of forums as I did research for this project. One night while looking for something completely unrelated I stumbled across a conversation that had an off-hand comment about the UPS that my ISP provides for some of their hardware. It turns out that the way this UPS is connected to the hardware, it only powered the phone functionality when it was running on the battery. That meant that all internet would be cutoff immediately when the power went out…even if there still was signal in the fibre lines. Prior to this little bit of information I had only intended on using a PDU in the rack. A quick trip to Amazon and I was expecting delivery of a 1U UPS that could provide battery backup to the UPS that provides battery backup to the ISP hardware. Because daddy can’t be without his internet when the power bumps.

Filled to overflowing

Our new house has a very nice wood burning fireplace in it, and I spent the better part of the winter sitting by it reading and researching all things networking. It was fun to spend time looking at concepts and problem areas that I used to work with early in my career. I’m not sure if it is the variety of new (to me) hardware and software, or the prospect of building v2 of something, but either way it is something that I was energized by.

Some of the other Western Devs have asked that I publish a bunch of content around this project, so this is just the first of many.

A while back I declared in a meeting that “Issues are cheap”. The context around the comment was such that I was suggesting that we can just create more issues when we need to surface a topic. If that issue goes away (closed, lost, whatever) and the topic surfaces again, create another issue. It’s cheap…all you have to do is a bit of typing.

My comment was met with the statement that, in fact, issues are not cheap. At first I was taken aback. My little world was being shaken by this statement. How could an issue not be cheap? It took mere minutes to create one. We don’t get charged for disk space or issue count by Github. The person that made that comment went on to clarify their position by saying that “Ideas are cheap”.

I forget what the rest of that meeting was about. My mind was racing to rationalize this distinction. In my world issues and ideas were one and the same. But in this person’s world they were clearly different. And it turns out they are right.

Ideas are the nascent concept that is represented within an issue. The issue is just the envelope that carries the message. Like a letter, the cost of writing a partially formed idea onto paper is trivial. A few minutes of your time, a pen and paper. This is cheap.

Unlike the writing on the paper however, envelopes have a lifetime cost associate with them. Mail needs to be picked up, transported, sorted, transported again, and delivered. Issues have this cost too. They must be triaged, prioritized, backlogged, re-prioritized, worked on, closed and retrospected on.

When you think about it from that standpoint, issues are really expensive. The thought that you can just create an issue doesn’t remove the fact that they will incur a cost throughout their lifetime.

Probably the most insidious type of issue is the drive-by one. You get an idea, you create an issue and you just keep right on moving. You don’t spend any time curating that idea, but instead push that workload to other people. Not only are you creating an expense for your process, but you’re asking other people to pay it off for you. I had a roommate like that once. I’m sure you know how that ended.

It’s important to remember that there can be hidden costs in the actions that we take. This was a bit of an eye opener to me. Having ideas isn’t a bad thing. Spending the time to make the well formed so that they carry value that warrants the effort that they will require as issues is a good first step.

I noticed that there was a lot of buzz around microservices and distributed systems at the conference this year. It seemed like there were endless conversations happening in the halls and at meals about the topic. While there were a lot of sessions that touched on the topic, or topics in the same space, there were a lot of areas that went untouched. For those that missed my sessions or those that are looking for a refresher here are the slide decks.

Slides

Additional materials

If you’re interested in diving deeper into microservices I have a Github repo that contains all the articles and videos that I’ve absorbed in the last year. If you find materials that are missing please send me a pull request. I’d love to read/watch more on the subject. https://github.com/dbelcham/microservice-material

Historically I’ve used a Microsoft Wireless Comfort Keyboard 5000 and it has served me well. Make no mistake, this is not an ergonomic keyboard. It simply has a few keys in the middle that are wider than normal to provide a bit of a spread between your hands.

Fellow WesternDev Simon Timms makes use of a clickety-clackety keyboard and swears up and down by it’s Cherry keys. Sure it makes him sound like he’s got a teleprinter machine pumping out the news in the back of his office, but for some reason the idea of a mechanical keyboard appealed to me. I don’t think I’ve had one since back in my 386 days.

After a bunch of research on ergonomic mechanical keyboards I settled on getting the Truly Ergonomic Keyboard (TEK). The only real decision you have to make when ordering is choosing between two models that have slightly different key layouts (double vs single keys in the lower corners of the keyboard).

The first two things that I noticed when the keyboard arrived were the size of it and the weight. The keyboard is narrower than the main keypad on the Comfort 5000 (note that the Comfort 5000 also has a number pad that the TEK doesn’t). The TEK is about 1.5 times heavier than the Comfort 5000 which not only makes it feel more substantial, it also makes it harder to push it around on your desk surface.

Getting started with the TEK is no more difficult than plugging in the USB cord. There’s no driver disk or software to install. Windows (10 in my case) recognized it straight away and I was off to the races….snail races that is.

Typing on the TEK takes a bit of getting used to if you’re a touch typist. The first touch typing issue that I ran into was the different orientation of the keys on the keyboard. On traditional keyboards, the main four rows of keys are arranged in columns that are angled from the bottom right to the top left.

On the TEK the columns of keys converge at the top middle of the keyboard, similar in pattern to an upside down V. What this means for your transition to the TEK is that you’ll constantly be reaching too far to the left when you move up from the home row and too far to the right when you move down from it. For me that meant that I was constantly reaching for the B key and, instead, getting the Backspace key (more about why it was the Backspace key shortly). There were other keys that I had similar problems on, but this was the most obvious since I was killing letters when I didn’t want to.

The second transition issue that I had was the location of many of the keys. Instead of having dead space in the middle like many ergonomic keyboards have, the TEK fills that space with a number of commonly used secondary keys. Windows | Command, Delete, -_, Backspace (which I just hit again while trying to type B) and Return all reside in the middle. When you first look at the keyboard it’s a novel layout. The Spacebar is split in two with the Return in the middle of them. The Shift and Ctrl keys also move up the keyboard by one row with Ctrl taking the places traditionally held by CAPS and Return. Getting used to Return being between the Spacebars was one of the tougher things for me at the start. I would type away and go to hit Return and get nothing….because I was actually pushing the Ctrl key. I have to say that having the Return key in the bottom middle of the keyboard between the Spacebars is a delight. I can’t imagine why this isn’t part of the standard layout.

The layout of the TEK does have some things that didn’t initially work for me. As a developer I tend to use some of the secondary and tertiary keys pretty frequently. For example { and } are almost as common as A and E. Those two keys are not located on the far right one row above the home row. Instead they’re on the complete opposite side of that row. I’m still working on getting fluent with typing those keys.

I also Tab a lot and I’m very used to using Alt+Tab and Ctrl+Tab to work my way through UIs. The Tab key is moved to the top left of the keyboard and is a single width key instead of the more traditional double width. Learning on a double width key made me very lazy in my finger placement when I was typing Tab. I learned fairly quickly to reach up a row to get to Tab but I couldn’t gain the pinky finger control to both reach up and accurately push the right key and this really affected the speed of my keyboarding when working in code. Luckily TEK has a solution.

The TEK is fully programmable. Simply create your layout on the website and save a file that it will generate for you. Flip a DIP switch on the back of the keyboard and flash the new programming onto it. Now I have a Tab key in the middle of the keyboard between the 5 and 6. Striking that 1.5x sized key with my index finger is much more accurate and fast. I reprogrammed the CAPS key so that it was reversed and Insert was the default with Fn+CAPS becoming the alternate. These are the only two tweaks that I’ve done thus far but it’s nice to have this option if I notice inefficiencies in the future.

So far I’m very happy with the Truly Ergonomic Keyboard. The mechanical keys are a dream to type on and the layout feels as if it is more efficient (my typing speed is about the same as on a more traditional keyboard even though I’m still working on muscle memory). Most importantly I haven’t yet noticed any hint of carpal pain which tells me that this keyboard is, at a minimum, no worse for me physically than what I was using before.

The biggest question you’re all probably asking is if it’s worth the (at the time of writing) $250 USD price tag. It’s hard to say. I’m not disappointed with having spent that kind of money on it, but each to their own.

All images provided courtesy of Truly Ergonomic

Over the last few months I’ve been seeing some intermittent stability issues. It seems that at the most random times the main router in the house will lock up and not respond to any pings. I have to then go downstairs, unplug the router for 30 seconds and plug it back in. By the time I’m back upstairs at my computer it’s all working again.

Thus far the disconnections haven’t happened during any meetings. When you work from home, having that happen would be a royal pain in the ass. As it is, going up and down the stairs during a train of thought is a complete pain in the ass so I’ve wanted to get to the bottom of it.

Logging in DD-WRT

By default logs for DD-WRT are stored in volatile memory. When you have to hard reset the router you, of course, lose those logs. This makes it really hard to track down the source of the problem.

An alternative is to log to the volatile memory location and then have a scheduled cron job to move those log files to an attached USB storage device. While this seems a bit better, you are never going to get the log entries that happened immediately before the router went down. There just isn’t time for the cron job to run and it’s on a schedule so it might not even be time for it to run.

A third option is to push the logging information to a remote server. Most examples out there show a configuration where the router is pushing the log entries to a computer on the internal network. This is all well and good, but I don’t want to have to guarantee that my desktop/laptop is running to receive and archive that data.

Buried in a wiki page on logging is mention of being able to log to a service called Papertrail. This got me thinking; I’ve logged from my Arduino and Netduino to logentries.com before. Why couldn’t I configure DD-WRT to do the same thing?

Configuration

There are three steps you need to do to get DD-WRT to log to logentries.com.

1. Setup a Log Set and Log on logentries.com
2. Configure DD-WRT
3. Restart DD-WRT

logentries.com

If you don’t already have one, create an account with logentries.com for their Free Plan. Once you’re logged into the website, open the Logs tab. Then click the “+ Add New” button and select “Add a log”.

The will put you into the page where you select the type of logging source you’re going to use. In the case of DD-WRT the logger is Syslogd so we select that one.

In the Configure screen you will want to add this Log to a Log Set. If you’re new to logentries.com you’ll want to select “New Set” and type a meaningful name in (i.e. Home Router). If you already have logs on logentries.com you may want to use an Existing Set.

The last step you need to take is to click on the “Create Log Token” button. This will open a bunch more information in a Configuration section of the same page.

The only thing you need to worry about at this time is what appears in the first command area. In my case it’s telling me to use *.* @@data.logentries.com:13630 which includes the endpoint information. This information is what you will need for configuring DD-WRT so don’t ignore it.

After you take note of the endpoint, you can click the “Finish & View Log” button. At this point your Log will be put into “Discovery Mode”. You have 15 minutes to make the first call to this endpoint to activate it. That call to the logentries.com endpoint needs to happen from the IP address of your router. This will configure the Log to only accept data from that IP address.

DD-WRT

Now that you have logentries.com configured you need to quickly move through the DD-WRT setup. Remember that you only have 15 minutes to get this done. It should be easy, but don’t wander off to the bowels of YouTube to watch cat videos.

First step, log into your DD-WRT router’s web interface and navigate to the Services tab.

Towards the bottom of that page you will find the System Log section. Enable Syslogd and enter the endpoint information that you got from logentries.com earlier. You will want to ignore the *.* @@ and only use the data.logentries.com:XXXXX portion, where XXXXX is the port that you were assigned.

Once you have this done, click Apply Settings followed by the Save button. After you’ve added the settings you can click the Reboot Router button and wait for your router to come back online.

Once your router has come back online and you have internet access you should be able to go to logentries.com and see that the “Discovery on Port XXXXX” has changed to show the IP address of your router. When I did mine it took a couple of minutes for logentries.com to process in the incoming messages and make this change. If you don’t see the IP address then you weren’t successful in connecting your router to logentries.com. I didn’t have the happen so the only suggestion I can give you is to verify that the Syslogd settings in DD-WRT were saved and then do a reboot of the router again.

Once you have successfully connected the two systems you should be able to open the Log for your router and see the logging that occurs when it goes through it’s startup process.

You could stop at this point and have all your system logging taken care of. But you can go one step further and enable firewall logging if you want. Note that when I tried this it made my router horrifically unstable and I had to completely disable it to get more than 5 consecutive minutes of uptime.
Open the Security | Firewall tabs and look to the bottom of the page. There, you will see a section for Log Management. Click Enable and set up the log levels and options as you desire.

Conclusion

Setting up DD-WRT and logentries.com isn’t very difficult. DD-WRT can be a bit persnickety but otherwise it’s a straight forward endeavour. Things you should note about this configuration:

• if your internet goes down you don’t get logging
• I’ve not had my router freeze up since doing this so I’m not sure what, if anything, I’ll capture
• if you only configure and run the Syslogd logging you will not get much activity in the log file

When my router craps out on me again I’ll post something about what I saw, or didn’t see, in the log files.

One of the things I’ve run into over the years is a need to change USD into CAD. The most common way people will deal with this is to go to their bank and make the currency exchange. The problem with this is that banks don’t give you a very good exchange rate. If the market rate is $1.30 CAD per $1 USD, the bank might only give you $1.27 CAD for each USD you give them. That’s a loss of $0.03 CAD by going to the bank vs going to the market. It might not seem like that much money, but if you’re exchanging $10,000 USD, going to the bank cost you $300 CAD. The more money you exchange, the more money you lose…and sometimes the spread is wider than $0.03.

So for those that don’t want to donate their money to banks that like to play the spread, what are our options?

• Bank exchanges
• Foreign exchange houses
• Norbert’s Gambit
• Market exchange

Banks

I’ve already talked about the process of exchanging money with banks. It’s not a good deal for you. Maybe its convenient to use them for a few hundred dollars once or twice, but over the long run (or large sums) you’re going to be losing more money than convenience provided.

Exchange houses

Foreign exchange houses operate in roughly the same way as the banks do. They make their money off the spread between market rates and the rate that they offer you. If you open a corporate account with them you’ll get “preferred” rates, which means that the spread is a bit smaller. In the end though, you’re getting into the exact same situation as you are with a bank.

Norbert’s Gambit

Norbert’s Gambit is a bit more complicated. To run the gambit you need to have a trading account that allows you to buy stocks on both a Canadian and US exchange. The way it works is that you buy a well traded (high trade volume) blue chip stock on the US exchange (usually NYSE). You then talk to the company you have your trading account with and get them to “journal” it over to the Canadian exchange (TSE). “Journaling” doesn’t sell the stock. Instead it converts it from the USD version to the CAD version. Journaling can take a day or two to happen depending on the company that you have your account with. Once you the journaling process has completed you can then sell the stocks on the Canadian exchange for CAD.

The theory behind this move is that the stocks on the different markets have the currency exchange built into them. When you journal the exchange rate is based on what the difference in price is for the stock on one exchange versus the other. So, for example, you might buy NYSE:RY (Royal Bank of Canada) at today’s closing price of $49.06 USD. Once the journaling process has finished you may own TSE:RY at $67.98 CAD. If you were to sell it at that point you’d be getting an exchange rate of $1.3856 (CAD for each USD). At the close of day today, the exchange rate on xe.com was $1.3846.

There’s a chance that during the period of time that it takes to journal the shares from one exchange to another the stock price will move in a way that isn’t in your favour, so there is some risk

VBCE

VBCE (Vancouver Bullion and Currency Exchange) operates in a very similar manner to any other exchange house. The difference is that it gives you spot pricing on the exchange you make. When you log into the website you request a quote. That quote, in my experience is almost identical to what you see on xe.com. For example, my first exchange with VBCE was at a rate of $1.444 while xe.com showed $1.447. Once you accept a quote then you fund the transaction. USD is sent from you to VBCE and CAD is sent back to you. Exchange complete.

VBCE allows you to fund your transaction via direct withdrawal and payment can be done via direct deposit. All you need to do is fill out the funding/payment web page with your pre-approved banking information and the process starts. The money movement, in my experience happens over a couple of days. First the payment of CAD to your account happens within 24hrs. The withdrawal of USD seems to take 1-2 business days. The nice thing about this is that you don’t have to go to a physical bank/exchange house nor do you have to place a call to do journaling. Currency exchange that you don’t have to wear pants for!

Because you accept a spot price before you do any of the money transfers the exchange rate that you accept is set set in stone. If the process of moving money takes a couple of days (or more if it’s over a weekend) you still get the accepted spot price. Currency fluctuation during between the time of the spot price and the final money deposit/withdrawal is of no concern to you. So from this standpoint you’re not only getting very close to the market rate, but you’re also protected from market movement.

Setting up a VBCE account wasn’t something that I just did online in 10 minutes. I needed to fill out a form that included banking references, talk with a representative on the phone and wait for VBCE to do their due diligence. The entire process took close to two weeks.

The process of sending paperwork and information to/from VBCE was not like anything I’ve dealt with before. I had to fill out paper copies of the forms and then scan them. The resulting pdfs and other required documents had to be zipped up and that zip file needed to be password protected. I then was instructed to email the zip file to them. A day later I got a phone call and we verbally exchanged the password to the file. Once my account was enabled I received instructions on how to make the transfer, my user name and my password. These came in a password protected zip file. Again, we verbally exchanged the password.

Now, I’m not pretending that password protected zip and pdf files are the end-all-be-all of security, but it was nice to see a company putting some thought into data security.

Fellow Western Dev, Simon Timms, suggested that he had similar experiences to me when dealing with VBCE. Simon ran into two issues during the process though. First, VBCE forgot to call him to get the password for the zip file containing his application. After getting that resolved and moving through the application process, Simon receive his instruction pdf without any password protection. So it seems like there is a bit of variance in how VBCE is applying its processes.

The only issue that I’ve run into with VBCE to this point has been that there is no online password reset process. If you need to reset you password (don’t judge my lack of memory…I forgot to enter it into my password software) you will have to call them and do it over the phone. Not a problem but you can only do it during business hours in the Pacific timezone.

Overall, after a couple of months using VBCE I’m pretty happy with my choice to use VBCE. It’s convenient and it saves me a bunch of money.

As I mentioned in my last blog post Microservices and Boundaries, I regularly see the question “How big should my microservice be?” The fast answer, albeit not the easy one, is that they should be the ‘right’ size. In that last blog post I talked about getting the right functionality into the right places (Antel for phone related functionality, Abitab for payment related functionality). There are a lot of people giving a lot of advise about scoping microservices, and I disagree with the majority of it. Here are some of the suggestions I’ve seen.

One Week’s Work

This scoping idea follows the premise that any microservice that you make should only take one week to write. There’s no discussion about the difficulty of the task at hand, the proper encapsulation, what the bounded context is or what the velocity of your development team is. Without fail, every one of your microservices should fit into a work week.

My question is what if the work isn’t done at the end of one week? I was once told that at that point you should throw it away and start over with a smaller scope. I’m not sure about you but to me that is about as wasteful as you could be. Run into a problem with a third party API that you can’t get resolved in a week? Scrap that work and start again.

Don’t think that what I’m saying is that we should allow our teams to take months to build microservices. We should be able to iterate on them quickly. This is one of the biggest benefits of taking the microservices approach. I think that the approach to writing your microservices should be one of Minimum Viable Product (MVP). If you write the bare minimum to have a functioning product, you will minimize the time spent on it. If the problem space is well bounded you will likely be working on microservices for very short periods of time. You shouldn’t, however, set an arbitrary timebox as a means to scope the work.

X Number of Lines

I can’t count the number of times that I’ve heard/read “your microservice should be no more than X lines of code.” Usually the value of X is absurdly small, even for the most terse programming languages. Common numbers I’ve seen thrown around are 250, 100 and 10. Yes, 10. Can you imagine trying to write a 10 line microservice that does everything that I explained that Antel was doing in my previous post? Of course not. You couldn’t even write the business logic for that process let alone all the infrastructure code that would be required (transactioning, logging, diagnostics) to create a healthy and sustainable microservice (more on this in a future post).

Long ago we decided that LoC was a horrible metric for measuring anything to do with software development. Why would we introduce it again? The number of Lines of Code required for a microservice will only be known once all the functionality has been written. Sure, you might be able to shrink that count with some judicious refactoring, but you’re not going to change 1500 lines of code into 100. If you can I’d suggest that you have a bigger cultural, or staffing, issue at hand.

Two Pizza Team

I hate this metric. I’ve seen it pop up in so many different discussions for so many different things. First, why pizza? Are we just a bunch of frat kids that haven’t let our culinary preferences evolve? Second, have you seen the size of pizzas in some countries (looking at you United States of Gluttony)? You could feed a batallion with two large pizzas. Third, what if you can’t decide on only two combinations of toppings? Sure you could get half-and-half done, but there’s only so combinations you can request before the pizza joint hangs up on you.

More importantly I’d like to ask what does this metric prove? Focus on technical and team structure concerns when you think about that. The only thing that I can think of is that it could cap the size of the team. What does that mean for code? Lower overall velocity? Maybe, but that is probably more influenced by the quality of the individuals on the team.

Summary

I’m a firm believer in working on microservices based on bounded contexts and minimum viable products. I don’t need some catchy phrase telling me how I need to staff up my team or plan my work. Instead I know that I need to invest time into understanding my problem and setting expectations on the work.

Just like microservices aren’t a panacea for software development, none of these are for sizing your microservices.

Here’s the process I went through to get data on a SIM.

First I went to an ANTEL store. ANTEL is the state owned telecom. It’s cheap, but like any good state company it has bureaucracy. I went into the store and the first stop is a concierge/secretary desk. Once I explained, through bad spanish and worse charades, what I wanted I was told that they would need my passport. Of course I didn’t have it so it was back to the hotel to get a passport. Back at the ANTEL store there was, of course, a new person manning the front desk so it was more bad spanish and Italian inspired charades. My request was understood and I was given a number and told to wait (Uruguay loves its number dispensers). About 5 minutes later I was called to a desk and the process of signing up begins.

After some furious keyboard smashing, and more horrible spanish on my part, I was told to go to the cashier to pay. I went across the lobby (really it’s all just one big room) and waited in line for the guy working the cashier wicket to finish with his mobile phone. I walk up and brutalize his native tongue. He looks at me, spins his chair around and pulls some paper off the printer. More furious keyboard smashing, a test of my rudimentary understanding of numbers in spanish and my payment was made. The hands me some paper and points me to another line and another wicket.

I line up once again and was called to the counter. I handed over the paperwork, the lady disappeared into the back room and re-appeared with the SIM card. A brief explanation of the phone number and security code and I was told I was done there. So I headed back over to the original gentleman that was helping me. At that point I was told that I was finished.

The SIM card went into the phone and I was connected to the ANTEL network…except I had no service. I couldn’t text, call or use data. I went back to the hotel where I had WiFi figuring that I needed to simply tweak some settings for the APN. An hour later I had changed the APN and still had nothing. At this point I gave up and called my friends at Kaizen Softworks. Within a minute I was told that I needed to go visit a place called Abitab. Once there we could charge our phone number with money. So off I go to the local Abitab (which I luckily knew about since I changed money at it earlier in the day).

I get to Abitab, go to a wicket, show them the paperwork with my phone number on it, tell them I want to put 400 pesos on the account and 3 minutes later I’m done. So now I have a SIM card, it has funds, but I still don’t have service. I needed to request a “plan” to use. Luckily when I was sitting in the hotel trying to figure out how to get service I found the magic code to do that. So I send an SMS to a predetermined number and boom….internet access.

There were a lot of steps in that process. More than what was needed some might say. If you think about it, there are good separation of concerns and lessons for microservices here. Let’s take a look at the process a different way.

ANTEL provides telecom services. I was a consumer (UI if you will) in need of those services so I reached out to their endpoints. From 40,000 feet this is a pretty standard microservice interaction. The first thing that the microservice required is that I authenticate myself (provide a passport). Once I’ve done that they start the process of creating a client record for me and putting in the purchase order for the SIM card. This is a pretty straight forward synchronous operation. In essence this is the code that runs in the controller of your REST WebAPI endpoint.

When the purchase order is successfully created anyone that cares is notified. In this case the cashier is notified by having a print out appear on his printer. I’m also notified that I need to go to a new location to do more ‘stuff’. In technical terms an event was published onto some kind of service bus. The cashier subscribed to that event type and printed a purchase order. The original service that I contacted provided a URI as part of its result to me that I needed to navigate to. That URI pointed me to the cashier service.

Once I made payment to the cashier service it published a successful payment event onto a service bus. The cashier service also provided me with a URI to visit to get the SIM card, which was the new line-up. The fulfillment service listened to that event and when I made my request to them they provided me with the SIM card and a bunch of meta data to go with it.

Like a good little consumer (UI) I made a call final call to an endpoint (the original person I dealt with) to check to see if my transaction was complete.

After dealing with ANTEL I moved on to Abitab, or the payment services. I called on one of their endpoints to make a payment. Again, I need to provide “proof” of who I am. In this case the level of proof is lower; I only need to provide the account number I want to deposit money to. The fact is, I can deposit money to any account I want and the service isn’t going to complain. It’s not hurting anyone but me if I do it wrong. So to make this payment I provided the amount of the payment, the means of payment and the account the payment should be applied to. I got a receipt. That concludes my interaction with the payment service.

Behind the scenes the Abitab service sent a message to some ANTEL service that notified ANTEL that the account needed to have credit applied to it and how much that credit should be. This is a service-to-service communication. Abitab knows nothing about the functionality of Antel. All Abitab does is publish a message that an event (payment) occurred. Abitab doesn’t need to know if, when, or how Antel applies the credit to the account so there is no need for a confirmation message so it can use a messaging pattern effectively here.

The separations of concerns in this case was very clear. One microservice (company) was responsible for all things phone related (SIM card, activation, etc) and another microservice was responsible for payments. If you’ve dealt with a system like this (Antel and Abitab) you’ll know that there are some significant benefits from this separation. First, and foremost, you can pay anything imaginable at Abitab. Need to pay your water bill? Go to Abitab. You TV bill? Go to Abitab. All Abitab does is take the payment and send a message to the company concerned tell them that a payment has been received.

When you’re writing microservices you need to look at the responsibilities and concerns of the services.  They need to be well encapsulated but also single-y responsible. If it feels like one microservice needs the functionality of another, look at ways to communicate between them. Ideally this communication should be unidirectional and in the form of commands. By taking this approach you will create microservices that do not effect each other in deployment, down(up)-time or changes.

I do almost all of my development work in Azure VMs these days. I can spin up a new one with Visual Studio already installed in minutes. This allows me to easily and quickly keep all of my different clients and projects in isolated buckets. So for one of my current projects I created a Windows 2012 R2 + Visual Studio 2015 VM. This is the VM that started throwing the above error. So when I asked the all knowing Google (and its smarter friend StackOverflow) I was perplexed by those three common solutions that I was finding.

1. Install Visual Studio 2010 – ummmm….no. Why should I need to do that?
2. Install the Windows SDK – there really isn’t one for Windows 2012.
3. Turn of generation of serialization assemblies – I have no idea if this project needs them or not so I’m not willing to make that change

This left me to piece together a solution on my own…which is likely why it took all afternoon. After looking through all the information that was buried in comments on answers for issues on StackOverflow, I managed to come up with this:

Open up regedit on your computer. You’re not going to change anything so just calm down. Navigate to Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\ToolsVersions\4.0

Note the SDK40ToolsPath entry and the value for it. This value will point to another registry key which you need to go find next. It’s likely going to point at something that starts with Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\

As you can see here there’s an InstallationFolder key that has a file path for you to go find in Windows Explorer.

When you get to that folder you’ll probably see what I saw above…no sgen.exe file. This is what the problem is. MSBuild is looking for sgen.exe in this location and can’t find it. How do you fix that? You could change registry settings, but there is risk in that…and I told you earlier that you wouldn’t have to. Another option is to find a copy of sgen.exe and put it in that folder. From what I can tell sgen.exe is pretty stand alone and any version you find should work. I went to a slightly newer SDK version on my machine and copied the file from there.

With sgen.exe now in the right folder, my build works just fine…so fine that removing the file keeps the build working and I wasn’t able to get a screenshot of the error for this post.

It doesn’t matter who you listen to or read, microservice isolation is going to be in the content. Why are they consistently bringing it up? Isolation is at the heart of microservices. A microservice is meant to be architected, created, deployed, maintained and retired without affecting any other microservice. You can’t do any, let alone all, of that without good isolation.

Databases

Probably the most common point made when talking about isolation is database sharing, or better stated, the idea that you should avoid it. Traditional monolithic application development usually sees that one large codebase working with one large database. Any area of the monolith can access any area of the database. Not only does the monolith’s codebase usually end up looking like a plate of spaghetti, so does the monolithic database. I can’t tell you the number of times that I’ve worked on brownfield codebases that have issues with data access, deadlocks being the most common. No matter how well factored a monolithic codebase is, the fact remains that the single database is an integration point for all the different moving pieces in that codebase.

To be able to release a microservice without affecting any other microservice we need to eliminate any integration that occurs at the database level. If you isolate the database so that only one microservice has access to it you’ve just said that the only thing that will be affected if the database changes is that one microservice. The testing surface area just shrunk for any of those changes. Another benefit is that have fewer pieces of code interacting with the database so you can, in theory, better control how and when that code does its thing. This makes it easier to write code that avoids deadlocks, row locks, and other performance killing or error inducing situations.

If you listen to enough people talk about microservices for a long enough time you’ll hear a common theme; 1 database per microservice. I’m going to disagree with the masses here and tell you something slightly different. You should have a minimum of 1 data store for each of your microservices. The difference is subtle but it’s important in my mind. There are times when you might want to store data in multiple different ways within one microservice. As an example you may be writing a Marketing  Campaign microservice. A RDBMS or noSQL database makes a lot of sense for storing the information about campaigns, content, targets, etc. But if you need to do statistical analysis of the campaign feedback (i.e. email bounces, unsubscribes, click-throughs, etc.) RDBMS and noSQL might not make the most sense. You might be better served with a data cube or some other type of data storage.

Is it going to be normal to have multiple data stores for one microservice? No…but you shouldn’t be worried if it does happen as long as you stay true to one thing: the microservice owns the data stores and no other microservices can access them.

Deployment Isolation

One of the primary goals of moving to a microservices architecture is that you’re able to deploy changes to one microservice without affecting any others. Additionally you want to ensure that if a microservice starts to fail it’s not going to bring down the other microservices around it. As such, this means that you’re going to need to have each microservice deployed in complete isolation. If you’re on a .NET stack you can’t share AppPools between them. If they do changes to the permissions or availability of that AppPool could (or likely will) affect other microservices. My experience with Apache is quite limited, but I’m sure there are similar concerns there.

One of the big current talking points around microservices is Docker. Building compartmentalized and deployable contiguous packages seems to address this goal. The only current issue is that Docker builds a smaller fence around the technologies that you can choose when solving your problems.  Docker, currently, doesn’t support Windows based applications. You can build your .NET apps and run them in a Linux Docker container, but that’s as close as you get…which might not be close enough for some “E”nterprise-y places.

Another piece of the deployment puzzle is what is commonly referred to as ‘lock-step’ deployments. A lock-step deployment is one where deploying one microservice requires a mandatory deployment of a different one. Usually this happens because the two components (or microservices in this case) are tightly coupled. Usually that coupling is related to api signature changes. I’m going to do a whole blog post on this later in the series, but its suffice to say for now that if you are doing lock-step deployments you need to stop and solve that problem before anything else. If you aren’t doing lock-step deployments you need to be vigilant to the signs of them and fight them off as they pop up.

Something that makes noticing lock-step deployments more difficult to notice, but is going to be mandatory in your deployment situation, is automation. Everything about your deployment process will need to be automated. If you’re coming from a mentality, or reality, of deploying single, monolithic applications you’re in for a big shock. You’re no longer deploying one application. You’re deploying many different microservices. There are a lot more deployable parts, and they’re all individually deployable. My project only had 4 microservices and we found that manual deployment was worse than onerous. Everything from the installation of the microservice to the provisioning of the environments that the microservice will run in has to be automated. Ideally you’re going to have automated verification tests as part of your deployment. The automation process gives you the ability to easily create frictionless lock-step deployments though…so you’re going to have to be vigilant with your automation.

I know that some of you are probably think “That’s all fine and good but I will always have to change APIs at some point which means that I need to deploy both the API and the consumers of the API together”. Well, you don’t have to…which kind of leads to…

Microservice <-> Microservice communication

At times there will be no way to avoid communication between two microservices. Sometimes this need to communicate is a sign that you have your bounded context wrong (I’ll be going over bounded contexts in a future post). Sometimes the communication is warranted. Let’s assume that the bounded contexts are correct for this discussion.

To keep microservices isolated we need to pay attention to how they communicate with each other. I’m going to do an entire blog post on this but because there can be so many things that come into play. It’s safe to say that you need to pay attention to a couple of different key pieces. First, take the approach of having very strict standards for communication between microservices, and loose standards for the technology implementations within each microservice. If you’re going to use REST and JSON (which seems to be the winning standard) for communication. Be strict about how those endpoints are created and exist. Also, don’t be afraid to use messaging and pub/sub patterns to notify subscribing microservices about events that happen in publishing microservices.

The second thing that you need to do, which is related to the first, is spend some time up front in deciding what your API versioning story is going to be. API versioning is going to play a big part in maintaining deployment isolation. Your solution will probably require more than simply ‘adding a version header’ to the communication. You’re probably going to need infrastructure to route communications to different deployed versions of the APIs. Remember that each microservice is an isolated deployable so there is no reason that you couldn’t have two or more instances (say v1 and v2) up and running at any one time. Consuming microservices can continue to make uninterrupted use of the v1 endpoints and as they have time/need they can migrate to v2.

Source code

Now that we’ve talked about the architecture of your microservices let’s take a look at the source code. So far we’ve been talking about isolating the microservice data stores, the deployment strategy and cross microservice communication. In all of those areas we didn’t come straight out and say it but each microservice is a separate application. How do you currently organize separate applications in your source control tool? Probably as separate repositories. Keep doing this. Every microservice gets its own repository. Full Stop. The result is that you’re going to see an increase in the number of repositories that you have to manage. Instead of one (as you’d probably have with a monolith) repository you’re going to have 1+N where N is the growing number of microservices that you’re going to have.

How you organize the pieces of the puzzle within that application is going to depend on many things. It’s going to depend on the components the microservice needs. The more “things” (service buses, web services, background processing, etc) then the more complicated the application’s source code structure is likely to be. You might have four or five assemblies, a couple or executables and other deliverables in a single microservice. As long as you have the minimum required to deliver the required functionality then I think you’ll be okay. More moving pieces can mean that you’re doing too much in the microservice though. It could be creeping towards monolith territory. So carefully watch how each microservice evolves into it’s final deliverable.

Another thing to consider is how you perform development isolation in your VCS. I’m not going to get into a branches vs feature-toggles discussion here, but you have to do something like that in your development practices. Things are going to move a lot faster when you’re developing, enhancing and maintaining microservices. Being able to rapidly turn around bug fixes, feature additions or changes becomes a competitive advantage. You need to work within your VCS in a manner that supports this advantage.

Continuous Integration

Following in the steps of the source code is the continuous integration process. Because every microservices is an independent application you’re going to need to have CI processes to support each and every microservice. This was one of the things that caught my project off guard. We didn’t see it coming but we sure noticed it when it happened. As we created new microservices we needed to create all of the supporting CI infrastructure too. In our case we needed to create a build project for every deployment environment that we had to support. We didn’t have this automated and we felt the pain. TeamCity helped us a lot but it still took time to setup everything. This was the first hint to us that we needed to automate everything.

Teams

There is a lot of talk about sizing microservices (which, again, I’ll cover in a future post). One of the things that continually seems to come up in that discussion is the size of teams. What is often lost is that teams developing microservices should be both independent and isolated…just like the applications that they’re building. Conway’s law usually makes this a difficult prospect in a ‘E’nterprise development environment. The change to developing isolated and compartmentalized microservices is going to require team reorganization to better align the teams and the products being produced.

Teams need to be fully responsible for the entirety of the microservice that they’re developing. They can’t rely on “the database guys to make those changes” or “infrastructure to create that VM for us”. All of those capabilities have to be enabled for and entrusted to the team doing the development. Of course these independent teams will need to communicate with each other, especially if there one of the teams is consuming the other’s microservice. I think I’ll have a future post talking about Consumer Driven Contracts and how that enhances the communication between the teams.

Summary

When talking about isolation and microservices many conversations tend to stop at the “one microservice, one database” level. There are so many other isolation concerns that will appear during the process of building, deploying and maintaining those microservices. The more I’ve researched and worked on microservices the more I’ve become of the opinion that there are a bunch of things related to microservices that we used to get away not doing on monolithic projects but that we absolutely can’t ignore anymore. You can’t put off figuring out an API versioning scheme. You’re going to need it sooner than you think. You can’t “figure out your branching strategy when the time comes” because you’re going to be working on v2 much sooner than you think.

Isolation is going to save you a lot of headaches. In the case microservices I’d probably consider leaning towards what feels like ‘too much’ isolation when making decisions rather than taking what likely will be the easier way out of the problem at hand.

What are microservices?

Sometimes its easier to start by describing what something isn’t. Microservices are not monolithic applications. That’s to say our traditional application was one unit of code. It is:

• developed together (even split as multiple assemblies, it was developed as one codebase)
• deployed together (the vague promise of “just deploy one dll from the application by itself” has never been practical or practiced)
• goes through the full application lifecycle as one contiguous unit (it is built as one, maintained as one and dies as one application)

So a microservice is none of these things. In fact, to start the definition of what a microservice is you’d be safe in saying that, at a high level, microservices are the opposite of all these things. Microservices are a bunch of small applications that represent the functionality we once considered as one application. They are:

• developed in isolation (contracts between microservices are established but this is the extent of their knowledge of each other)
• deployed in isolation (each microservices is it’s own encapsulated application)
• lives and dies no relationship to any other microservice (each microservice’s application lifecycle is managed independently)

There’s a pretty common comparison between microservices and SOA. If you missed the whole SOA bandwagon then 1) you’re younger than me, 2) I’m envious of you, and 3) it had its merits. Some people will say “microservices are just SOA done right”. I’m not sure that I fully agree with that statement. I don’t know that I disagree with it either. As an introduction to microservices you should understand that there are many parallels between them and SOA. Applications that aren’t monolithic tend to be distributed. Both SOA and microservice architectures are decidedly distributed. Probably the biggest difference between SOA and microservices is that SOA was quickly absconded by the big software manufacturing companies. According to them, doing SOA right meant using an Enterprise Service Bus (most likely an Enterprise Service Broker being marketed and pitched as a service bus)…and preferably using their ESB, not a competitors. Gradually those SOA implementations became ESB implementations driven by software vendors and licensing and the architectural drive of SOA was lost. Instead of a distributed system business logic was moved out of application codebases and into centralized ESB implementations. If you’ve ever had to debug a system that had business logic in BizTalk then you know what the result of vendors taking over SOA was.

Thus far (and microservices are older as an architecture than you probably think) the microservice architecture hasn’t been turned into a piece of software that companies are flogging licenses for. It’s questionable whether the hype (justified or not) around Docker as a core component of a microservices implementation is just the start of microservices heading the same direction as SOA. It might be, it might not be.

Each microservice is a well encapsulated, independent and isolated application. If someone pitches microservices at you and there’s a shared database, or two microservices must be deployed in unison, or changes to one microservice forces changes on another then they’re pitching you something that’s not a true microservice. These concepts are going to lead to a bunch of the items I will discuss in future blog posts.

What does all of this mean?

Good question. The short story, from what I’ve been able to collate, is that microservices promise a bunch of things that we all want in our applications; ease of deployment, isolation, technology freedom, strong encapsulation, extensibility and more. The thing that people don’t always immediately see is the pain that they can bring. Most organizations are used to building and managing large monolithic codebases. They struggle in many different ways with these applications, but the way that they work is driven by the monolithic application. As we covered earlier, microservices are nothing like monoliths. The processes, organizational structures and even technologies used for monolithic applications are not going to work when you make the move to microservices.

Think about this: how does your team/organization deal with deploying your monolithic application to a development environment? What about to Dev and Test? Dev, Test and UAT? Dev, Test, UAT and Prod? I’m sure there’s some friction there. I’m sure each one of those environment deployments takes time, possibly is done manually, and requires verification before its declared “ready”. Now thinking about all the time you spend working through those environment releases and imagine doing it for 5 applications. Now 10. What would happen if you used the same processes but had 25 applications to deploy? This is what your world will be like with microservices.

Part of what I hope to convey in this series of blog posts is a sense of what pain you’re going to see. There’s going to be technical pain as well as organizational pain. And, as I know from experience over the last 6 months, there is going to be a learning curve. To help you with that learning curve I’ve created a list of resources as part of a github repository (https://github.com/dbelcham/microservice-material) that you can go through. Its not complete. It’s not finished. If you find something you think should be added please submit a pull request and I’ll look at getting it added. As of the moment I’m writing this it’s probably weakest in the Tooling, Techniques and Platforms section. I’m hoping to give that some love soon as it will be important to my writing.

In closing

Microservices are a topic that is gaining traction and there’s a lot of information that needs to be disseminated. I don’t think I’m going to post any earth shattering new concepts on the topic but I want to get one location where I can put my thoughts and, hopefully, others can come for a cohesive read on the topic. I am, by no means, an expert on the topic. Feel free to disagree with what I say. Ask questions, engage in thoughtful conversation and tell me if you think there’s an area that needs to be covered in more depth.

The first stage of doing a proof of concept is getting a user account set up. When I tried registering a new account with plivo I got the following message:

I did, however, receive an account confirmation email. I clicked on the link in the email and received the same message. Thinking that this might just be a UI issue and that the confirmation was accepted I decided to try to login. Click the login button and wham…same error message…before you even get to see the username and password screen. This, obviously, is starting to become an issue for our proof of concept. I decide, as a last resort, to reach out to plivo to get a conversation going. I navigate to the Contact Us page to, once again, see the same error message before I see the screen. At this point the website is obviously not healthy so I navigate to the status page and see this

So everything is healthy….but it’s not. A quick glance at their twitter account shows that plivo attempted to do some database maintenance the day prior to this effort and they claimed it was successful. Related to the database maintenance or not, I needed to move on.

This is the part that gets interesting (for me anyways). Our choice in SMS provider was a commodity selection. We went the store, looked on the shelf, read the boxes and picked one. It very well could have been any box that we selected. But the fact that making that selection was so simple means that changing the selection was equally simple. We weren’t (in this specific case which may not always be the case) heavily invested in the original provider so the cost of change was minimal (zero in our case). All we had to do was make a different selection.

This highlighted something that hadn’t clicked for me before. Software as commodities makes our developer experience both more dynamic and more resilient. We are able to change our minds quickly and avoid unwanted interruptions easily. The flip side of the coin is that if you’re a SaaS provider you need to have your A game on all the time. Any outage, error or friction means that your potential (or current) customer will quickly and easily move to a competitor.

There are a number of different ways that you can sharpen edged tools (chisels and hand planes). There are machines, you can use water stones, Arkansas stones, diamond stones or, my personal choice, the “Scary Sharp Technique”. For those of you that couldn’t be bothered click that link and read through the original usenet posting on the topic in detail (and who can blame you, this is a software blog after all) here’s the TL;DR; for Scary Sharp.

• Sharpening is done with wet dry automotive sand paper, not stones
• Progression is made to finer and finer grits as you get sharper. i.e.  400 –> 800 –> 1200 –> 2000 grit
• Sandpaper is glued down to a perfectly flat surface such as float glass, a tile, or granite counter top (ask the missus first if you’re planning on using the kitchen)

My station for tonight looked like this (400 grit to the left, 2000 grit on the far right):

So, why am I boring you with all this detail about sharpening my chisels? There’s a story to be told about software tools and woodworking tools. The part of it that I’m going tell in this post is the part about maintaining and fine tuning them.

Maintaining your tools

For me to be able to effectively, and safely, create my next woodworking project I need to constantly maintain my chisels (amongst other tools). I have to stop my project work and take the time to perform this maintenance. Yes, it’s time that I could be getting closer to being finished, but at what cost? Poor fitting joinery? Avoidable gouges? Self amputation? The trade off is project velocity for project quality.

Now think about a development tool that you use on a regular basis for your coding work. Maybe it’s Visual Studio, or IntelliJ, or ReSharper, or PowerShell, or…or…or. You get the point. You open these tools on an almost daily basis. You’re (hopefully) adept at using them. But do you ever stop and take the time to maintain them? If it weren’t for auto-updating/reminder services would you even keep as close to the most recent release version as you currently are? Why don’t we do more? I currently have an install of Resharper that I use almost daily that doesn’t correctly perform the clean-up command when I hit Ctrl-Shift-F. I know this. Every time I hit Ctrl-Shift-F I cuss about the fact it doesn’t work. But I haven’t taken the time to go fix it. I’ve numbed myself to it.

Alternatively, imagine if you knew that once a week/month/sprint/<timeframe of your choosing> you were going to set aside time to go into the deep settings of your tool (i.e. ReSharper | Options) and perform maintenance on it. What if you smoke tested the shortcuts, cleaned up the templates, updated to the latest bits? Would your (or mine in the above example) development experience be better? Would you perform better work as a result? Possibly. Probably.

Tools for tools

I have tools for my woodworking tools. I can’t own and use my chisels without having the necessary tools to keep them sharp. I need a honing guide, sand paper, and granite just to be able to maintain my primary woodworking tools. None of those things directly contribute to the production of the final project. All their contributions are indirect at best. This goes for any number of tools that I have on my shelves. Tools for tools is a necessity, not a luxury. Like your first level tools, your second level of tools need to be maintained and cared for too. Before I moved to the Scary Sharp system I used water stones. As you repetitively stroke the edge over the stone it naturally creates a concave in the middle of the stone. This rounded surface makes it impossible to create a proper bevel angle. To get the bevel angle dead on I needed to constantly flatten my stone. Sharpen a tool for a while, flatten the stone surface…repeat. Tools to maintain tools that are used to maintain tools.

Now think about your software development tools. How many of you have tools for those tools? Granted, some of them don’t require tools to maintain them….or do they? So you do some configuration changes for git bash’s .config file. Maybe you open that with Notepad ++. Now you have a tool (Notepad++), for your tool (git bash). How do you install Notepad++? With chocolatey you say? Have you been maintaining your chocolatey install? Tools to maintain tools that are used to maintain tools.

Sadly we developers don’t put much importance on the secondary and tertiary tools in our toolboxes. We should. We need to. If we don’t our primary tools will never be in optimal working condition and, as a result, we will never perform at our peaks.

Make time

Find the time in your day/week/sprint/month to pay a little bit of attention to your secondary and tertiary tools. Don’t forget to spend some quality time with your primary tools. Understand them, tweak them, optimize them, keep them healthy. Yes, it will take time away from delivering your project/product. Consider that working with un-tuned tools will take time away as well.

First thing I had to do was set up an account at logentries.com. It’s easy to do and it gives you 30 trial days of the full suite of features before reverting back to a free tier. There are a lot of different options for setting up logs once you’re in the system. At first I wanted to try sending log entries via a REST endpoint since it was what I knew best from my previous work with the Arduino. logentries offers a REST endpoint (HTTP PUT), but it’s being deprecated. So I looked at the other raw API options; Plain TCP/UDP and Token-based.

Plain TCP/UDP

This type of endpoint is all about sending TCP and/or UDP packets containing your log entries to an endpoint. The tricky thing with it is that it ties you to a specific IP address as the source of your log entries. The trick is that there is a 15 minute window which links the incoming message and it’s source IP address to your account’s log. Not a horrible thing, but a setup restriction none-the-less. More information on how it works can be found here.

Token-based

Like the Plain TCP/UDP option, all log traffic is sent to an endpoint via TCP and/or UDP. The difference is that your log data will contain a token and that will be used to tie the messages you send to the account you’re using. This is much easier than worrying about getting the right IP address linked to the account in the first 15 minutes of the log’s life. I chose this option because of the simplicity. More info on it here.

Arduino code

I’ve been playing with sending data from my Arduino to Azure Mobile Services (more on that another time) over the last few weeks. As a result I had some pretty good code written (mostly borrowed from Adafruit examples) to hook up my Arduino to WiFi. I bought a CC3000 shield from Adafruit for my Uno and made use of it for this project too. There’s a lot of boilerplate code to get the WiFi up and running, but that’s not the interesting part. What you’re here to see is how the data is sent to https://logentries.com/logentries.com after a connection has been established.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

void Log(char logEntry[100]){
  t = millis();
  if (!logger.connected()){
    Serial.println("Connecting logger");
    do {
      logger = cc3000.connectTCP(log_ip, 80);
    } while ((!logger.connected()) && ((millis() - t) < connectTimeout));
  }
  Serial.println("logger connected");

  if (logger.connected()) {
    Serial.println("logging");
    logger.fastrprint(LOG_TOKEN);
    Serial.print(LOG_TOKEN);
    logger.fastrprint(" ");
    logger.fastrprintln(logEntry);
    Serial.println(logEntry);
    logEntry[0] = 0;
  }
}

There are a couple of things going on in this method.

1. On line 6 we establish a TCP connection to the logentries.com endpoint (data.logentries.com).
2. If the connection to the endpoint is made successfully we move on to sending the log entry. Lines 13, 15 and 17 make this happen. “logger” represents the connection to the endpoint and we call .fastrprint(….) to send data and .fastrprintln(….) to send data with a line ending.

All we need to send out in our stream to the endpoint is the log data we want to include and the token we got when we created the log on the logentries.com website. By sending the token as a .fastrprint, the blank as a .fastrprint, and the message as a .fastrprintln we’re essentially sending all three of these pieces of information as one line to the endpoint.

The output

Here’s what things look like when you look on the website.

Now you can start making use of logentries’ tagging and reporting functionality to understand how your Arduino code is working.

Source code for the sample project is available on github: https://github.com/dbelcham/logentries_arduino

Enter our garage door system…the LiftMaster 8355

We have a keypad outside for entry to the garage and by the door into the house there is a multi-function control panel that allows us to turn on the system’s lights and open/close/stop the door. From what I gather most garage door systems traditionally have had a ‘short-to-trigger’ style system which allowed you to easily interface with them and kick off the movement of the door. Chamberlain (the underlying manufacturer of this system) did away with that and has implemented a proprietary communication between the keypad and multi-function control. From what I can figure out the communication across the wire is serialized so you can’t simulate another device.

That’s okay though because Chamberlain has created something that they call MyQ. This is a system that requires you to purchase a ‘hub’ that enables wireless communication between phone apps and a website (hosted by Chamberlain) and your garage door. Here’s what the system looks like conceptually.

So yah…there are no moving parts in there what-so-ever. There is no architectural way around this. The wireless protocol from the LiftMaster to the MyQ hub is proprietary and unknown. There is some speculation that it is a modified implementation of Z-Wave, but nobody seems to know speculate more than that. The communication between the MyQ Hub and the Chamberlain servers is also proprietary, although I think I could sniff the wires to see what is being sent. Certainly the calls between the phone apps and your browser can be sniffed and someone has created an unofficial api based on some of that work. That api doesn’t help the overall architecture any. The two main points in the architecture are the MyQ hub (which is an additional $129 US investment that you must make) and the Chamberlain servers.

If the Chamberlain servers go offline (for maintenance or otherwise) the whole system goes offline. You can be sitting 5 feet from the MyQ hub clicking furiously on your phone app and the door will not move. Granted, you could walk over and push the multi-function panel button, but that’s not the point here. If you can’t get data service from your smart phone, neither the phone nor the website will be of use to you since you can’t get to either. The problem with these is that both of these known points of failure that you and I have no control over. The success of our “smart home” experience is being entrusted to someone else.

Since you’re here at my blog, you’re likely a developer, so let me put this to you another way. Today nuget.org experienced an outage. Not a “server is down” outage. Not a “internet is unavailable” outage. It was an outage caused by a bad DNS record. Remember when Azure had that happen? Not cool right? Nuget.org being down today was one of those things that probably really annoyed a whole bunch of developers. It likely didn’t stop them from doing some work, but it was another burr under the saddle. The whole architecture implemented by Chamberlain has the same potential. DNS issues, routing issues, maintenance, etc…they can all become a burr under the saddle of our smart home experience. I’ve had my share of burr under the saddle moments and as I get older I’d like to have fewer.

To me that means eliminating moving parts. The first thing I want to do is drop the reliance on the Chamberlain servers. That takes a lot of the possible fail points out of the equation. Ideally I’d like to drop the MyQ Hub. It’s $129 expense and it doesn’t add any additional value to the smart-home experience beyond enabling communication to the LiftMaster. That leaves two options:

1. direct access to the wireless protocol used to communicate between the LiftMaster 8355 and the MyQ Hub
2. direct connectivity (short-to-trigger) at the LiftMaster

Today I took to twitter to ask Chamberlain why they don’t want to enable option #1 as a direct API. Ironically their twitter handle is @chamberlaindiy….yet you can’t DIY anything with this system.

@dbelcham We’re integrating with several tech partner systems, including @Nest and @TheWinkApp. Also part of the HomeKit smart home platform
— Chamberlain-DIY (@ChamberlainDIY) September 3, 2014

@dbelcham At Chamberlain, safety & security are our top priority. We’d be happy to discuss further offline if you’d like.
— Chamberlain-DIY (@ChamberlainDIY) September 3, 2014

So they think that keeping things proprietary and in a tight circle of integrators is the best thing since “…safety & secruity are our top priority”. Essentially they’re promoting security through obscurity which is commonly accepted as one of the most dangerous things you can do with a security system. Think about the security issue a bit more practically though. Is the garage door the biggest attach surface that is available on your house to a criminal? Is it the easiest? I’d answer no to both of those questions and argue that the windows and exterior doors offer more options for entry and that those options are easier than hacking your garage door opener. Even if you did hack the garage door, and get it open, an attached garage will usually have an exterior grade door between it and the house. Maybe I’m a chronic door locker, but I treat that door like any other exterior door when it comes to locking it. So really, at a practical level, the only thing that you’re trying to protect by having proprietary communications is whatever is in the garage. That’s the worst case scenario…a poorly implemented device/system using the API (much like was exposed at Black Hat 2013 on the proprietary Z-Wave home automation protocol). Heck, that can happen right now if one of Chamberlain’s integration partners do a poor job of implementing the proprietary MyQ protocol.

@chamberlaindiy reached out to me to have a conversation offline about my concerns so hopefully sometime in the near future that will happen and maybe…..just maybe….I can convince them that opening up a well written communications api would be in their best interest.

If you’re interested in my Circuit Breaker implementation you can find it at https://github.com/dbelcham/dervish

I’ve dabbled on the IT Pro side of the fence in the past so I know that cobbling together a wiring solution was likely to end up in a world of pain. Rather than do that I decided that one of the first things that needed to be done to the house was to add some wiring in. As with many “stock” houses these days, the only wiring done was coax to 3 locations for TV and some Cat5e run for phones. Everything terminated at the power panel with hardly any extra cable to use. The way that our house is setup, the power panel is in an awkward location that didn’t lend well to much of anything. There was no way I was building an accessible wiring closet in that location without completely blocking the power panel so I had to come up with another option (plus, the area near the power panel, awkward as it is, was quickly designated the future wine cellar…priorities right?).

The location that I ended up settling on was about 15ft away from where all the builder installed wiring was terminating. So I had a couple problems on my hands. First, I needed to get more wiring run, and second I needed to extend the current cabling to this new location.

Extending cabling

As I said earlier I had Coax and Cat5e that needed to be extended. The Coax isn’t so hard. there are plenty of male-to-male connectors available. All you need to do is buy/build some cables to get from the current endpoint to the desired endpoint. Because I hate sloppy cabling I decided to custom make my cables so that they fit precisely to length. A few tools, 30 minutes in Home Depot watching a YouTube video on how to crimp ends on and I was good to go. Because I didn’t want the cables running willy-nilly under our stairs I spent some time with a speed bore and punch holes in the studs so that I could run the Coax through the walls and have it pop out right where I needed it.

The Cat5e extension was a bit more of an issue. There really aren’t that many ways to extend network cable. I did manage to find a set of 110 punchdown boxes though. Wire goes in one side and punches down; wire comes out the other side where it was punched down. A small PCB board in the box makes all the connections for you. So, like the Coax, I custom cut some Cat5e, ran it through the studs and ended it where my new termination location was going to be.

Running more wire

Most new houses are built with no data wiring in them. It seems that the common belief is that WiFi is ubiquitous and convenient enough that there’s no value in doing so. I disagree. WiFi is convenient. It is easy to procure and setup. It doesn’t, however, offer a very good data transfer rate for most devices. Yes 802.11N offers decent speeds but its nothing compared to gigabit and if the WiFi signal is weak in an area of the house, the data connection will be slower than advertised. On top of that, not all the devices in our house are WiFi enabled so they either have to sit near the router or be off the network. Neither of those options will work for us here. And don’t get me wrong, there will be WiFi in the house.

So to fill my need for gigabit speed I got some electricians to stop by and run a bunch of Cat6 cables for me. Each run ends in the basement termination location. Here’s what I ended up getting:

• 4 runs to the upstairs TV area (more on why 4 in a future post)
• 4 runs to the main floor TV area
• 1 run to each of two different office areas
• 1 run to the laundry room (again, more in a future post)

Unlike the home builder, I had the electricians leave 20+ feet of extra cable on the termination end just in case I changed my mind about the termination location. Most of that extra cable, once trimmed, has gone to building patch cables so it wasn’t a waste at all.

If you have the chance do get data cabling done before the walls in a house are closed up…or get lots of empty conduit run. We were lucky that all that cabling only required one hole in an inner closet wall. Not much damage but the time spent getting the cables pulled sure added to the cost.

Termination point

Once I had cables run to a centralized location I needed to figure out what I was going to do to manage the termination of these feeds. After some googling around I found out about media enclosures. These are just a metal box that gets installed between two wall studs and gives you a solid platform to mount different devices to. I had a bunch of small devices that I wanted to house so this seemed like a good idea. In the end it is home to my 8-way coax splitter, 8-way phone junction point, cable modem, cable-to-phone box and a WiFi router (more on that in a later post too).

I waffled on the idea of terminating the long runs in this box. I knew it wasn’t the cleanest or most flexible solution but for something like the coax lines I likely wasn’t going to be changing their configuration ever so in the end simplicity won out. All of the Coax runs terminate in the enclosure. None of the data cables or phone runs go to the media enclosure. The connection between the cable modem and the WiFi router stays contained in the enclosure and a single data run leaves the WiFi router and the enclosure to connect to a gigabit switch. The same is true for the cable-to-phone box; all of its connections are kept in the enclosure and only 4 cables from the phone junction point exit the enclosure. In the end there is 1 Coax cable into the enclosure and 1 Cat6, 4 Cat5e and 3 Coax out of the enclosure.

Now, I needed to manage the termination of the 12 or so data lines and the 4 phone lines that I had coming to this central location. Unlike the coax lines flexibility in configuration was going to be a huge benefit here. To that end I ran all of those cables into a rack and terminated them in a patch panel. I also terminated the line from the WiFi router in the patch panel. This give me the ability to directly connect any data line wall jack in the house directly to the internet connection. I can also create a physically separate network if I need/want to. Right now all required data lines are patched from the patch panel to the switch giving full gigabit connectivity within the house.

Extending WiFi

I hate weak WiFi signals. To combat this I put a WiFi router (configured as an access point) on each floor and connected it via one of the gigabit data runs back to the main switch. With that I actually killed two birds with one stone: I was able to get stronger WiFi everywhere, and I got a 4 port switch in those locations. The 4 port switch actually turned out to be very useful. At one TV location all of the following devices can be connected:

• Xbox
• WDTV Live Hub
• TV
• HD cable box
• DVD player (this one lost out as we rarely use it)

The end configuration

In the end it all logically looks something like this

And, amazingly enough, it all works as desired and with all of this I now have a foundation to add home automation components onto.